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Abstract: Assessment surveys of students are often conducted in order to evaluate online learning activities. Most 
surveys measure responses to questions which are based on students’ subjective impressions. The purpose of this 
study is to examine participants’ assessments made during the transitional phase in an online learning environment 
which includes blended and fully online courses at a Japanese national university. Students were enrolled in 
two-unit Master’s or Bachelor’s degree courses which were taught by the same professor. The total number of 
students with valid survey data was 184 (92 Masters, 67 Bachelors for the blended learning course and 25 
Bachelors for the fully online course). A survey questionnaire consisting of 10 questions measured the 
self-assessments of students’ online learning experiences. Three factors were extracted. There are no significant 
differences in all factor scores between the beginnings and the ends of the courses. These results show the 
coherence of students' assessments during the course. The correlation coefficients of the first factor scores 
(e-learning evaluation) between the beginnings and the ends of the courses are not high, however (Masters: r=0.35, 
Bachelors for blended learning: r=0.46, and Bachelors for fully online: r=0.33). Therefore, some participants have 
changed their evaluations between the two surveys. When the differences in factor scores from the initial and final 
surveys are compared between students who rated the course highly at the beginning (high raters) and students 
who did not (low raters), the scores for the high raters decrease and the scores for the low raters increase. Also, the 
relationships between students’ transitions and the metrics of their behaviour were investigated. 
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1. Introduction 

The use of online learning environment is spreading widely in the curricula of modern university 
courses, in accordance with the development of information technology and online communications 
(Twigg 2001). When these technologies are introduced into the learning environment, learning 
performance and effectiveness can be discussed as a cost benefit (Bates 1999). Since it is well known 
that it is not easy to emphasize the cost benefit of a new technology or a new educational system, 
determining the impressions and level of satisfaction of stake-holders is often preferred to surveying 
participants’ opinions, according to the first stage of the Kirkpatrick model which suggests that they be 
observed and measured (Watkins et al. 1998). 

Therefore, assessment surveys of students are usually conducted in order to evaluate progress in the 
learning environment. Most surveys measure responses which are based on a students’ subjective 
impressions. In particular, as the online learning environment is very different from the conventional 
learning environment, both students and teachers have to adapt their abilities. These assessments help 
to improve the teaching material, course content and supporting methodologies (Harrington & Reasons 
2005; Anderson et al. 2006). Additionally, learning skills and student's attitudes should be observed 
throughout the learning process, and appropriate survey items should be developed to help design 
support programs for both students and teachers (Craig et al. 2008). Recently, these evaluation results 
have been used formally, for the purpose of assuring the quality of instruction. The role of assessment 
has been well recognized as an important management activity (Deepwell 2007). In these scenarios, 
both students’ satisfaction surveys and assessing the level of student’s academic performance are 
major indices, and can be used as measures of teaching quality at universities (Elton 1993). 

Also, the issue of quality assurance is becoming more important in Japanese higher education, as 
teaching quality and improvements in methodology are major topics of concern for global universities 
(Marginson 2006). To provide learning opportunities for university students, the online learning 
environment is an appropriate tool (Kaneko 2009). The differences between students' and teachers' 
recognition of their roles in online education were measured over two years using an annual survey 
(Palmer and Holt 2009). According to the experiences of students and teachers, the results of students' 
evaluations have changed across three years (Nakayama et al. 2009). 

As academic terms are not short, student's perceptions and skills of adaptation may change during the 
course. Their attitudes toward the learning environment may develop week by week, as conventional 
psychological phenomena such as the primary effect and the Hawthorne effect are often observed 
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(Haebara et al. 2001). The subjective evaluations of the learning environment by those in a distance 
education system were significantly different between the initial and final class sessions (Shimizu and 
Maesako 1988). The online learning environment is evaluated using factors which include material 
evaluations and self assessment (Nakayama et al. 2006); these evaluations may change along with the 
progress of learning, and also may be affected by various metrics, such as the number of class 
sessions attended. These transitions in subjective evaluations may be related to learning performance 
and other metrics of students’ characteristics, which were extracted during previously conducted 
studies (Nakayama et al. 2008, 2009). 

The purpose of this study is to examine participants’ transitional phase assessments in a blended 
learning environment and in a fully online learning environment at a Japanese national university. Also, 
the effectiveness of these transitions on learning performance is investigated. This is a small scale 
case study, however, because courses differ from university to university. In this study, the 
determination of the student's change in evaluation during a course is focused on initially, using 
conventional statistical analysis of the survey responses. 

2. Method 

2.1 Survey group 

2.1.1 Courses and participants 

A survey was conducted using two-unit 15-week Master’s and Bachelor’s courses, which were taught 
by the same professor at a Japanese national university. These two credit courses, which were 
organized as blended learning, were offered during the 2006 and 2007 Spring Terms. One course was 
"Information Society and Jobs", a Bachelor-level class for university freshman, and the other one was 
"Advanced Information Industry Studies", a Master's degree course for students in their first year of 
graduate studies. Most of the students were Engineering majors. The third course was "Information 
Society and Jobs", the same 2-unit Bachelor-level class for university freshmen, which was offered as a 
fully online course in 2006 and 2007. Students could choose to attend either the blended or the fully 
online course, in accordance with their preferences. Since freshmen in Japanese universities are busy 
attending various other courses, the fully online course can provide a flexible style of learning, if 
students are self motivated. Therefore, it is clear that fully online learning requires participants to 
possess a degree of information literacy and time management skills. 

The total number of students with valid survey data was 184 (92 Masters; 67 Bachelors for blended 
learning, and 25 Bachelors for fully online learning). 

2.1.2 Online co urses 

For the blended learning course, students attended face-to-face classes, and were also able to access 
the course content online outside of class. This online learning material was designed for a fully online 
course, and also for a blended learning course, to encourage students to maximize their learning 
experience. The materials, which were created in advance, consisted of lecture videos, slides and 
online tests which allowed students to check the progress of their learning achievement themselves. In 
particular, these online tests offered unlimited trial tests for students, and this system recorded the 
number of trials and the final test scores. 

To encourage maximum participation in e-learning, in particular for the blended learning course, an 
explicit benefit was provided to students: online module test scores would count towards their final 
grades in the course. Also, students could make up for class absences by taking and passing online 
tests that corresponded with the face-to-face class sessions which were missed. This encouraged 
students to participate in online modules and tests, because if they missed a regular face-to-face class 
session it often affected their final test scores and the evaluation of their learning experience. Most 
students paid careful attention to their performance and final grades. 

All students took part in the final test, which consisted two essay tests at the end of the course. 
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2.2 Survey instruments and data 

All classes were surveyed using the same questions and constructs used in our earlier surveys 
(Nakayama et al. 2006). Several surveys of participants were conducted, and the results of these have 
already been reported (Nakayama et al. 2009). In this paper, we will focus on self-assessment of the 
online learning experience. To explain the differences in responses to questions, various indices were 
analyzed such as students’ information literacy and learning performance, and their essay writing 
ability. 

2.2.1 Learning experience 

The construct used to measure students' online learning experiences consisted of a 10-item Likert-type 
questionnaire. This construct was originally developed by the authors to assess student’s activity during 
a blended learning course. It has been used repeatedly and its validity, including the factor structures 
has been confirmed over three years. The questions are shown in Table 2. The questionnaire, which 
asked about student’s overall impression of the online course, their own learning habits, and their 
learning strategies, required each item to be rated using a 5-point scale. The surveys were first 
conducted during the second week of classes and were then conducted again at the ends of the 
courses. 

2.2.2 Information literacy 

Information literacy as a characteristic of students was measured using a construct (Nakayama et al. 
2008). Fujii (2007) has defined and developed inventories used to measure information literacy. For this 
construct, the survey consisted of 32 question items, and 8 factors were extracted: interest and 
motivation, fundamental operating ability, information collecting ability, mathematical thinking ability, 
information controlling ability, applied operating ability, attitude, and knowledge and understanding. 
This construct was surveyed during the second week of the courses. 

Secondary factor analysis was conducted on the scores of the above eight factors which were 
calculated using the survey data. As a result, two secondary factors were extracted (Nakayama et al. 
2008). The first secondary factor (IL-SF1) consists of “operational confidence and knowledge 
understanding”; the second one (IL-SF2) consists of “attitude issues”. 

2.2.3 Learning performance 

Some indices regarding learning performance during these courses were measured. Three indices 
were used as indicators of learning performance: the number of days attended (NDA), the number of 
completed modules (NCM), and the online test scores (OTS). 

For Master’s students, the final test was conducted as a reporting-style essay. Master’s students wrote 
summary reports which were selected from two out of 5 topics. For Bachelors, the final test was 
conducted with a proctor during the final exam period assigned by the university. All Bachelor students 
gathered in a lecture room, and wrote answers to four questions. Two questions included 
multiple-choice tasks and the other two were essay tests. 

The essay tests were evaluated by experts using an automated system (Nakayama et al. 2009). First, 
the essay tests were assessed by two outside experts. They independently evaluated essays using a 
3-point scale (0-2) which used 5 criteria: certainty, fitness for learning content, argument, various 
aspects and drawing illustrations. The sum of 5 scores were used as an expert assessment. Second, 
for the assessment of the essay, an automated scoring system (Ishioka and Kameda, 2003) was used. 
It is possible to use this system via a web site. As a result, another score which was calculated using 
assessment software, consisted of three factors: "rhetoric", "logical structure" and "content fitness". 

All data were analyzed to extract differences in survey timings using simple statistical tests, such as 
f-tests, correlation analysis and factor analysis (Coolican 1994). 
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3. Results 


3.1 Learning activities 

To compare the learning activities of Master’s and Bachelor students taking online courses, indices of 
learning, rates for the number of days attended (NDA), and rates for the number of completed modules 
(NCM) are summarized in Table 1. This table shows mean rates and STDs of indices, and the mean 
difference between Master’s and Bachelor students in the blended learning environment is tested 
statistically using a f-test, in order to make a clear comparison. The results and f-values are indicated in 
Table 1. There are significant differences in NCMs between Master’s and Bachelor students (p<0.01), 
while there are no significant differences in NDAs. According to our previous analysis (Nakayama et al, 
2006), Master’s students prefer online modules more than Bachelors do, and this result confirms the 
tendency. Rates of days attended were quite high, so there was no difference in this measurement for 
Master’s and Bachelor students. 


As a reference, the rate of the number of completed modules (NCM) for Bachelors in the fully online 
learning course is indicated in Table 1. The rate is comparable with the rate for the blended learning 
environment. 


Table 1: Summary of learning activities 



Master 

Bachelor 

Bachelor(FO' 


(N=92) 

(N=67) 


(N=25) 

N of days attended (NDA) 

0.95 

0.95 

t(157)=0.7 

N/A 


(0.08) 

(0.06) 



N of completed modules (NCM) 

0.91 

0.84 

t(153)=2.7** 

0.86 


(0.17) 

(0.15) 


(0.10) 

(): STD, **: significant level p<0.01, FO: Fully Online course 



Table 2: Question items and mean scores for Master’s and Bachelors (blended and fully online) 
students at the beginnings and the ends of courses 



Master(BL) 

5achelors(BL 

iachelors(FC 

Question items 

1 


F 

1 


F 

1 


F 

Q1. E-learning is easy to follow and understand 

## 


ii ii 
## 

## 


ii ii 
## 

## 


## 

Q2. 1 learn better in an on-line course 

## 

« 

## 

## 

< 

## 

## 


## 

Q3. On-line materials are useful to me 

## 


## 

## 


## 

## 


## 

Q4. It is easy to schedule on-line learning time 

## 


## 

## 


## 

## 


## 

Q5. On-line course content is interesting 

## 


## 

## 

> 

## 

## 


## 

Q6. Overall, on-line course is a favorable learning experien 

## 


## 

## 


## 

## 


## 

Q7. I'm a conscientious student 

## 


## 

## 


## 

## 


## 

Q8. It is my habit to do course preparation and review 

## 


## 

## 

> 

## 

## 


## 

Q9. 1 have my ow n method and w ay of learning 

## 


## 

## 


## 

## 


## 

Q10.1 have my ow n strategies on how to pass a course 

## 


## 

## 

< 

## 

## 


## 


Significant level «: p<0.01; <: p<0.05; I: Initial, F: Final, BL: Blended 


Learning, FO: Fu 


ly Online cou 


3.2 Mean values for question items in transition 

Mean values for the 10 question items reveal some differences in means between the beginnings and 
the ends of the courses, and are summarized in Table 2. To determine the changes in responses, 
statistical tests such as f-tests of the initial and final mean scores have been conducted and the 
significant differences are marked with "<" symbols according to the level of significance. 
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For example, for question item Q2 (“I learn better in online courses”), the means of both Master’s and 
Bachelor students in the blended learning environment increased significantly from the beginnings to 
the ends of the courses. 

This result suggests that most participants prefer to learn using online materials along with the course. 
On the other hand, means for Q5 and Q8 decreased at Bachelor student levels in the blended learning 
course. This means that some problems have occurred with Bachelor students in the blended learning 
environment. A support system should be provided for these students because they are freshmen and 
do not have experience with the learning environment. 

There was no significant difference in means between the beginning and the end of the fully online 
course, however most means of assessment were higher than the means of assessment for the 
blended learning course. Participants in the fully online course were satisfied with the learning setting 
and recognized the benefit of the course. 

The results coincided with the results of our previous report (Nakayama et al. 2007), and the tendency 
for the responses to be the same was validated. 

3.3 Factor score transitions 

The model for the 10 survey questions, which had been used previously, consisted of three factors 
(Nakayama et al., 2009). Again the factor structure from the analysis of 456 participants across three 
years (Nakayama et al., 2009) was used. The contribution ratio and correlation across factor axes is 
summarized in Table 3. For this survey, the factor structure was used to determine students' attitudes 
toward temporal transitions. 


Table 3: Correlations between factor scores 



LI 

L2 

L3 

Q1. E-learning is easy to follow and understand 

0.70 

-.03 

0.10 

Q2. 1 learn better in an online course 

0.48 

0.20 

-.08 

Q3. Online materials are useful to me 

0.61 

0.10 

-.10 

Q4. It is easy to schedule online learning time 

0.57 

-.16 

0.18 

Q5. Online course content is interesting 

0.68 

0.11 

-.05 

Q6. Overall, online course is a favorable learning e> 

0.82 

-.07 

-.02 

Q7. I'm a conscientious student 

-.04 

0.51 

0.07 

Q8. It is my habit to do learning preparation and rev 

0.11 

0.49 

0.08 

Q9. 1 have my ow n method and way of learning 

-.01 

0.18 

0.62 

Q10. 1 have my ow n strategies on how to pass a cc 

0.01 

-.01 

0.65 

Correlation amongst factor axes LI 

1.0 



L2 

0.40 

1.0 


L3 

0.21 

0.39 

1.0 

Contribution ratio by each factor ignoring other facte 

0.28 

0.14 

0.12 

Total contribution 


0.43 



LI: e-Learning overall evaluation; L2: Learning habits; L3: Learning strategies 


The factor scores in this survey are estimated using means of marked scale values for factor items. 
Here, the focus is on the initial and final factor scores. The factor scores for Master’s and Bachelor 
students for blended and fully online courses at the beginnings and ends of courses are summarized in 
Figure 1. As the figure shows, there are no significant differences in mean scores between the initial 
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and the final scores of courses using Mests, while there is a significant difference in the third factor 
scores between Master’s and Bachelor students for blended learning (Nakayama et al., 2007, 2009). 
This result confirms the coherence of participants' evaluations of their learning experiences for both 
blended and fully online courses. 


The course was conducted across 15 weeks, and participants felt strange about recording their 
impressions of their progress during the course. A detailed analyses was conduced as follows. 
According to the procedure for factor analysis, factor scores correlate with each other partially because 
the structure of the three factors is extracted as a Promax solution (Nakayama et al., 2009). The 
correlational relationships in this survey were confirmed, as shown in Table 3. This result suggests that 
the second factor scores correlate with both the first and third factor scores. 


Initial 

FI: e-Learning evaluation 
Final 

Initial 

F2: Learning habits 
Final 

Initial 

F3: Learning strategies -1 
Final 


Master 

Bache lors(Blended) 
Bachelors 
(Fully Online) 



T 

3 4 

Factor score 


Figure 1: Comparison of factor scores between Master’s and Bachelors (blended and fully online) 
students at the beginnings and the ends of courses. 


3.4 Relationships of factor scores between the beginnings and the ends of courses 

To determine the coherence of factor scores during a course, factor scores of the relationships were 
analyzed. Figure 2 shows a scatter gram, which illustrates pairs of the first factor scores between the 
beginnings and ends of the Master’s and Bachelor (blended and fully online) courses. The regression 
lines were superimposed over the figures for Masters and Bachelors blended and fully online courses 
respectively. There were some differences in slopes of the three regression lines. 

These superimposed figures suggest that factor scores between the beginnings and ends of courses do 
not coincide. Therefore, individual factor scores deviate during the courses. 

On the other hand, the relationships between factor scores for Factor 2 is illustrated in Figure 3. Since 
scores of Factor 2 are calculated from responses to two items, the plots are sparse. Most plots are 
gathered along a diagonal line, and both scores are correlated, though some deviations can be seen in 
the figure. The regression lines almost overlap each other without much difference in the angles of the 
slopes. 
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A Bachelors(Fully online) 


2 3 4 5 

Initial Factor-1 score 


Figure 2: Scatter gram of factor 1 scores between the beginnings and the ends of courses. 



Figure 3: Scatter gram of factor 2 scores between the beginnings and the ends of courses. 
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To confirm this tendency, correlation coefficients of factor scores between the beginnings and the ends 
of the courses are summarized in Table 4 for Master’s students, in Table 5 for Bachelor students in the 
blended course and in Table 6 for Bachelor students in the fully online course. The vertical cells show 
the factor scores at the beginning (I: initial), and the horizontal cells show the factor scores at the end (F: 
final). For the three tables, diagonal coefficients show the degree of deviation of the factor scores. As all 
coefficients between the second factor scores are above 0.7, this assessment aspect may be deemed 
to be stable. For the third factor score, the coefficient for Bachelor students in the blended course 
(r=0.37) is relatively small because many freshman Bachelors have acquired new learning strategies 
during the course, though a number of Bachelors who had already become information literacy experts 
participated in the fully online course instead of the blended course. For the first factor score in all 
Master’s and Bachelor classes, the coefficients are relatively small (Master: r=0.35, Bachelors in 
blended: r=0.46, Bachelors in fully online: r=0.33), so student’s assessments may have changed during 
the courses. 


Table 4: Correlation coefficients of factor scores between the beginnings (I) and the ends (F) of courses 
(Master’s) 


Master (Blended) 

F1(F) 

F2(F) 

F3(F) 

FI: e-Learning overall evaluation(l) 

0.35 

0.24 

0.15 

F2: Learning habits(l) 

0.20 

0.72 

0.34 

F3: Learning strategies(1) 

-0.01 

0.32 

0.51 


Table 5: Correlation coefficients of factor scores between the beginnings (I) and the ends (F) of courses 
(Bachelor: Blended) 


Bachelors(Blended) 

F1(F) 

F2(F) 

F3(F) 

FI: e-Learning overall evaluation(l) 

0.46 

0.22 

0.15 

F2: Learning habits(l) 

0.43 

0.70 

0.46 

F3: Learning strategies(l) 

0.25 

0.33 

0.37 


Table 6: Correlation coefficients of factor scores between the beginnings (I) and the ends (F) of courses 
(Bachelor: Fully online) 


Bachelors(Fully Online) 

F1(F) 

F2(F) 

F3(F) 

FI: e-Learning overall evaluation(l) 

0.33 

0.34 

0.30 

F2: Learning habits(l) 

0.25 

0.71 

0.45 

F3: Learning strategies(l) 

0.44 

0.61 

0.55 


When observing correlation coefficients across the three factor scores, the coefficients between the first 
and the third factor scores are small for both Master’s and Bachelors students in the blended course but 
not for the fully online course. As shown in Table 3, the original factor scores are weakly correlated as 
well. In Tables 4, 5 and 6, the initial second factor scores correlate significantly with all final factor 
scores except for the first factor in the fully online course. The initial second factor scores show the 
extent of participant’s learning habits, and all factor scores increase during the courses when 
participants have acquired some additional learning habits, though the second factor axis correlates 
with the axes of the other two factors. Also, students whose learning habits are poor should be 
encouraged to improve them and be supported in doing this. 
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3.5 Students’ transitions during courses 

According to the above analyses, assessment transitions may provide some critical information when 
individual differences in factor scores are extracted and correlation coefficients are calculated between 
the differences in factor scores at the beginnings and ends of courses. These coefficients are 
summarized in Table 7. 

All coefficients between the differences in the initial factor scores are negative values. This result shows 
that when the initial factor scores are low the differences are large. Again, Figure 2 shows some low 
raters in the initial survey who received high scores in the final survey. As the absolute value of the 
coefficients is large, this tendency is obvious in the upper panel of Table 7. Examples of the largest 
coefficients are the e-learning evaluation factor scores (FI) for Bachelor students in the fully online 
course (r=-0.65), and the learning strategy factor scores (F3) for Bachelor students (r=-0.60). All factor 
scores for Master’s students are almost always large, such as r=-0.60, -0.47, and -0.54. 

Table 7: Correlation coefficients between factor scores and differences in factor scores 



Score difference 


FI 

F2 

F3 

Initial: Master(BL) 

-0.60 

-0.47 

-0.54 

Initial: Bachelors(BL 

-0.37 

-0.34 

-0.60 

Initial: Bachelors(FC 

-0.65 

-0.21 

-0.44 

Final: Master(BL) 

0.54 

0.28 

0.44 

Final: Bachelors(BL 

0.65 

0.43 

0.52 

Final: Bachelors(FC 

0.46 

0.61 

0.49 


All coefficients are significant: p<0.05 except w here 


The coefficients between the differences and the final factor scores are also calculated, and all 
coefficients are positive and almost always high because when the final factor scores are high, the 
differences are large. Therefore, high factor scores come from final scores which have increased since 
the initial scores. 

When survey assessments of the online learning environment are conducted, the timing of the survey 
may affect the results. Some participants become discouraged during the course, but the assessor will 
still receive a good assessment at the end of the course if evaluations come from satisfied students 
whose impressions have become more positive during the course. If the timing of the survey is at an 
earlier stage in the course, the structure of the responses may be the opposite. 

3.6 Relationship between students’ transitions and other metrics 

To examine the relationship between students’ transitions and other metrics, participants who have all 
metrics were selected. Therefore, the number of students decreased to 78 Master’s and 45 Bachelors 
students for blended learning, and 25 Bachelors for the fully online course. 
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First, the impact of student’s information literacy (IL) on students’ transitions and on the assessment 
factor scores is confirmed. For Master’s students in the blended learning environment, both information 
literacy, “operational confidence and knowledge understanding” (IL-SF1) and “attitude issues” (IL-SF2) 
affect both initial and final assessment factor scores. There was no relationship between information 
literacy and differences in factor scores though some correlational coefficients changed between the 
initial and the final surveys. Some negative coefficients appeared between the first factor (FI) and 
learning performance, such as NDA and OTS, because the indices of learning performance of 
participants decreased in accordance with degree of their preference for online learning (FI). There 
were no significant relationships for other factors. Also, performance in the essay test did not affect the 
factor scores of students’ transitions. 

For Bachelor students in the blended learning environment, the first information literacy measure: 
“operational confidence and knowledge understanding” (IL-SF1) affected factor scores of students’ 
transitions. There are some positive coefficients between learning performance and students’ 
transitions contrary to Master’s students. Most positive coefficients suggest that scores for “learning 
habits” correlate with the number of days (NDA) of face-to-face sessions which were attended, and 
scores of “learning strategies” correlate with the number of completed online modules (NCM). The 
difference in the first factor positively correlates with the NCM (r=0.32), since these participants have 
recognized the benefits of online learning. Positive evaluation of online learning (FI) may influence 
essay scores, experts assessment scores (EXP) and automated evaluation scores (Auto scores), 
because the coefficients are negative. 


Table 8: Correlational relationship between transitions of factor scores and learning performance 
(Masters) 


Masters (Bl 

FI 

Initial 

F2 

F3 

FI 

Final 

F2 

F3 

Difference 

FI F2 F3 

IL-SF1 

0.22 

0.40 

0.27 

### 

0.28 

### 

### 

### 

### 

IL-SF2 

0.23 

0.48 

0.23 

0.20 

0.49 

0.29 

### 

### 

### 

NDA 

### 

### 

### 

#### 

### 

### 

#### 

### 

### 

NCM 

### 

### 

### 

### 

### 

### 

### 

### 

### 

OTS 

### 

### 

### 

ii ii ii ii 

zzzzzzzz 

THTTTTr 

### 

### 

ii n ii ii 

ztztnn 

ii ii ii ii 

### 

### 

EXP score 

### 

### 

### 

### 

### 

### 

### 

### 

### 

Auto score 

### 

### 

### 

### 

### 

0.22 

### 

### 

### 


IL-SF1: operational confidence and knowledge understanding, IL-SF2: atl 
NDA: N of days attended, NCM: N of completed modules, OTS: Online te 
EXP: Expert essay assessment score, Auto: Automated essay assessme 


For Bachelors students in the fully online course, there are a few positive coefficients between 
information literacy and factor scores of students’ transitions. Because the differences in factor scores 
between the beginning and the end is small and the number of valid participants is not large, the 
number of significant correlational coefficients is small. During online courses, learning activities may 
not be affected because there are no face-to-face sessions and no collaborative learning activities 
amongst fellow students. According to this hypothesis, students’ transitions may be due to collaborative 
learning activities where instructors and fellow students have collaborated during face-to-face sessions. 

The confirmation of this hypothesis will be a subject of our further study. 
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Table 9: Correlational relationship between transitions of factor scores and learning performance 
(Bachelors: Blended learning) 



Initial 

Final 

Difference 

Bachelor (BL 

FI | F2 | F3 

FI | F2 | F3 

FI | F2 | F3 

IL-SF1 

### 

### 

### 

### 

### 

### 

### 

### 

### 

IL-SF2 

### 

### 

### 

### 

### 

### 

### 

### 

### 

NDA 

### 

### 

### 

### 

### 

### 

### 

### 

### 

NCM 

### 

### 

### 

### 

### 

### 

### 

### 

### 

OTS 

### 

### 

### 

### 

### 

### 

### 

### 

### 

EXP score 

### 

### 

### 

### 

### 

### 

### 

### 

### 

Auto score 

### 

### 

### 

### 

### 

### 

### 

### 

### 


IL-SF1: operational confidence and knowledge understanding, IL-SF2: at 


NDA: N of days attended, NCM: N of completed modules, OTS: Online tes 
EXR Expert essay assessment score, Auto: Automated essay assessrr 


Table 10: Correlational relationship between transitions of factor scores and learning performance 
(Bachelors: Fully online learning) 


Bachelor (FO’ 

FI 

Initial 

F2 

F3 

FI 

Final 

F2 

F3 

Difference 

FI F2 F3 

IL-SF1 

0.11 

0.43 

0.34 

0.21 

0.30 

0.27 

0.05 

-0.05 

-0.07 

IL-SF2 

-0.03 

0.37 

0.29 

-0.03 

0.34 

0.34 

0.00 

0.05 

0.07 

NDA 

- 

- 

- 

- 

- 

- 

- 

- 

- 

NCM 

0.11 

-0.35 

-0.16 

-0.04 

-0.04 

-0.29 

-0.13 

0.32 

-0.15 

OTS 

-0.14 

0.31 

0.17 

-0.03 

0.10 

0.09 

0.11 

-0.19 

-0.08 

EXP score 

0.25 

0.15 

0.38 

0.09 

0.24 

0.33 

-0.16 

0.15 

-0.04 

Auto score 

0.07 

0.12 

0.23 

0.34 

0.15 

0.29 

0.20 

0.07 

0.08 


IL-SF1: operational confidence and knowledge understanding, IL-SF2: attitude issue 
NDA: N of days attended, NCM: N of completed modules, OTS: Online test score 
EXP: Expert essay assessment score, Auto: Automated essay assessment score 


4. Conclusion 

In this paper, assessments of participants' transitions were analyzed to determine the effectiveness of 
online learning which included blended learning and fully online courses. The mean factor scores, 
which were extracted as a measure of learning experience, remained at the same levels between the 
beginnings and the ends of the courses. The individual assessment scores for e-learning evaluations 
changed dramatically, though the scores for learning habits were relatively stable. The differences in 
factor scores during the courses correlate positively with the final scores, while these differences 
correlate negatively with the initial scores. Additionally, several indices of learning performance were 
surveyed, and there were some relationships between a number of these indices and participants’ 
transitions in the blended learning course, while there were few such relationships in the fully online 
learning course. 

Therefore, the course assessors should bear in mind the timing of the survey and the significance of the 
results. The process of transition during the course should be analyzed more closely. Also, some 
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effective support procedures for participants need to be developed. A detailed discussion of these items 

will be the subjects of our further study. 
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